3D touch based user interface value pickers

ABSTRACT

Electronic devices that include a force sensor input and input user interface elements are described. The force sensors may be located to detect force on the display of the electronic device. The force sensor, alone or in combination with one or more other sensors such as capacitive touch sensors, allows for interaction with the user interface input on the device. By using a hold detection logic with a pressure level sensitive sensor, user interface elements can be manipulated or values can be assigned to input elements.

FIELD OF TECHNOLOGY

The present invention relates to the technical field of user interfaces.In particular, the invention relates to 3D touch based user interfaces.

BACKGROUND OF THE INVENTION

A large and growing population of users employs various electronicdevices to perform functions, such as placing telephone calls (voiceand/or video), sending and receiving email, text messaging, accessingthe internet, playing games, consuming digital content (e.g., music,movies, images, electronic books, etc.), and so on. Among theseelectronic devices are electronic book (eBook) reader devices, mobiletelephones, desktop computers, portable media players, tablet computers,netbooks, and the like.

Many of these electronic devices include touch screens to allow users tointeract with the electronic devices using touch inputs. While touchinput is an effective way of interfacing with electronic devices in someinstances, in many instances touch inputs are problematic. For example,it may be difficult to use touch inputs when using a device one handed.As another example, when interacting with a device using touch input, auser's finger or stylus typically obscures at least a portion of thescreen.

Additionally, many existing electronic device interfaces are cumbersomeand/or unintuitive to use. User interface controls used to performvarious functions are often esoteric and unintuitive to users. Forexample, UI Pickers are elements of user interfaced used for pickingfrom the end user a value with well-defined type and range. Thesepickers are often part of a mobile UI that requires an input from theuser. One example of such a picker is a number picker which evolved froma simple text field with numerical validation into a virtual spinner.This virtual spinner requires interaction with the user to bring up theinterface and further interaction to correctly select the desired value.

While picking or sliding the control has become much simpler than typingin the value in a mobile device there is still no use of the newlyemerging technology of 3D/Force touch screens. Thus, there remains aneed for new interfaces for electronic devices and techniques forinteracting with such interfaces and electronic devices.

SUMMARY OF INVENTION

The present invention, in an embodiment, relates to an apparatus forinterfacing with a computer device comprising a display comprising aforce sensor, a force sensor controller configured to determine a forceinput based on a touch input to the display, and a processor coupled tothe force sensor controller and configured to receive the force input,determine a user interface element associated with the location of theforce input, determine a consistent hold has occurred, and modify theuser interface element based on the force input. In an optionalembodiment, modifying the user interface element based on the forceinput comprises applying a numerical quantity to an input field of theuser interface element. In a preferred embodiment, the numericalquantity comprises a binned quantity. In an example embodiment, the userinterface element comprises a picker. In a further example embodiment,the user interface element comprises a scroll input. In a preferredembodiment, wherein determining a consistent hold has occurred comprisesconverting the force input into a binned force quantity and determiningthat the binned force quantity has not changed over a period of time. Inan optional embodiment, wherein the period of time is between 0.5seconds and 2 seconds.

In an alternative embodiment, an electronic device comprises a display,at least one sensor to detect a user interaction event with the display,and a processing system to interpret data from the sensor and to displaya user interface element on the display, wherein, the sensor receives atouch input from a user and generates a digitized force value and adisplay touch location, and wherein the processor receives the digitizedforce value and display touch location from the sensor, determines ifthe user interface element has been activated based on the display touchlocation, and if activated, determines if a consistent hold hasoccurred, and if a consistent hold has occurred, applies a data input tothe user interface element. In a preferred embodiment, the data input tothe user interface element comprises a numerical quantity. In a furtherpreferred optional embodiment, the numerical quantity comprises a binnedquantity. In an optional embodiment, the user interface elementcomprises a picker. In another optional embodiment, the user interfaceelement comprises a scroll input. In an preferred embodiment, whereindetermining if the consistent hold value has occurred comprisesconverting the digitized force value into a binned force quantity anddetermining that the binned force quantity has not changed over a periodof time.

A method for modifying computer device user interfaces comprisesreceiving, from a display with an integrated force sensor, a touchinput, determining, by a controller operatively coupled to the display,a display location and a force measurement based on the touch input,mapping, by a processor coupled to the controller, the display locationto a user interface element, determining, by the processor, that aconsistent hold has occurred, and applying the force measurement to theuser interface element. In a preferred embodiment, applying the forcemeasurement to the user interface element comprises inputting aquantitative value to an input field of the user interface element. Inan optional embodiment, the user interface element comprises a picker.In another optional embodiment, the user interface element comprises ascroll input. In a further optional embodiment, applying the forcemeasurement to the user interface element comprises scrolling the pagebased on the force measurement. In a preferred embodiment, determining aconsistent hold value has occurred comprises converting the forcemeasurement into a binned force quantity and determining that the binnedforce quantity has not changed over a period of time. In an optionalembodiment, the period of time is between 0.5 seconds and 2 seconds.

Numerous other embodiments are described throughout herein. All of theseembodiments are intended to be within the scope of the invention hereindisclosed. Although various embodiments are described herein, it is tobe understood that not necessarily all objects, advantages, features orconcepts need to be achieved in accordance with any particularembodiment. Thus, for example, those skilled in the art will recognizethat the invention may be embodied or carried out in a manner thatachieves or optimizes one advantage or group of advantages as taught orsuggested herein without necessarily achieving other objects oradvantages as may be taught or suggested herein.

The methods and systems disclosed herein may be implemented in any meansfor achieving various aspects, and may be executed in a form of amachine-readable medium embodying a set of instructions that, whenexecuted by a machine, cause the machine to perform any of theoperations disclosed herein. These and other features, aspects, andadvantages of the present invention will become readily apparent tothose skilled in the art and understood with reference to the followingdescription, appended claims, and accompanying figures, the inventionnot being limited to any particular disclosed embodiment(s).

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and the invention may admit toother equally effective embodiments.

FIG. 1 illustrates a mobile device having a 3D touch UI, according to anembodiment of the present invention.

FIG. 2 illustrates a process for a 3D touch UI, according to anembodiment of the present invention.

Other features of the present embodiments will be apparent from theDetailed Description that follows.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the following detailed description of the preferred embodiments,reference is made to the accompanying drawings, which form a parthereof, and within which are shown by way of illustration specificembodiments by which the invention may be practiced. It is to beunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the invention.Electrical, mechanical, logical and structural changes may be made tothe embodiments without departing from the spirit and scope of thepresent teachings. The following detailed description is therefore notto be taken in a limiting sense, and the scope of the present disclosureis defined by the appended claims and their equivalents.

This disclosure describes, in part, new electronic devices, interfacesfor electronic devices, and techniques for interacting with suchinterfaces and electronic devices.

For instance, this disclosure describes example electronic devices orcomputers that include a force sensor input and a processor orprocessing elements. In some embodiments, at least one of the forcesensors may be located at or near the display of the electronic device.Using force sensor alone or in combination with one or more othersensors, such as capacitive touch sensors, in some embodiments, theelectronic device allows for simple one-handed or hands-free interactionwith the device. For instance, the device may include one or moreinertial sensors (e.g., gyroscopes or accelerometers), or the like,information from which may be integrated with information from the forcesensor for sensing gestures performed by a user or other combinedinteractions.

This disclosure is addressing a UI element being presented on a devicewhich has the hardware of determining the level of pressure that oneprovides. This can be a tap on a mobile device screen, using digital penor force touch trackpad installed on a laptop. In embodiments, it alsorequires an operating system that provides an API to capture such touchevent and the level of pressure. The hardware to determine the level ofpressure may contain a controller that generates a digitized force valueor other related data. One such example is an iPhone 6s device with iOSoperating system running on it. However, although widely supported iniOS devices this technology can also be found on other device fromdifferent vendors running operating systems such as Android.

FIG. 1 illustrates a system 100 having a 3D touch UI, according to anembodiment of the present invention. The display 120 of the device 110includes a force sensing element 130. The display 120 may also show aninput user interface element 140. The input user interface element 140takes a numerical input from the user. The input user interface element140 may, in embodiments, be a UI picker.

By the term UI picker, user interface elements that their intention isto pick a value from the end user are referred to. The followingdescription demonstrates implementing the essence of using a numberpicker, in an embodiment.

FIG. 2 illustrates a process 200 for a 3D touch UI, according to anembodiment of the present invention. In step 210, a touch is detected bythe mobile device. In step 220, a force measurement is obtained. In step230, the user touch is mapped to a UI element and the force measurementchanges the UI element displayed value. The user interface element maybe activated based on the user touch. In step 240, the system determinesthat the force measurement has been held. In step 250, the UI element'svalue is adjusted based on the force measurement.

The implementation of the number picker takes advantage of a touch eventthrown by the operating system to the application view controller forthe picker when a user touch is detected. The user touch may be mappedto the particular UI element. The picker controller is able to accessthe following information through touch events.

1. The location on the screen where the touch event occurred. Thecontroller handles events only occurred within the graphical area of thepicker.

2. The maximum allowed force as it may vary between types of hardware,although fixed for a specific hardware (predetermined by the system, notuser-specific)

3. The force value of the current touch event.

4. The normal force value for an average touch with no intention toprovide force (predetermined by the system, not user-specific). In iOS,for instance, the information for location, maximum and force are allpart of the event object thrown by the iOS to the view controllers.

The range of numbers that can be picked by the picker must be relativelynarrow. Such a narrow range can be mapped into the range between normaltouch force value and maximum available force value. The number ofelements within the numbers range can then determine the force valueranges for each number.

The range of force value for each picker element can be calculated byusing the following calculation.

Define MAX as the maximum possible force value and it is a constantprovided by the operation system. For instance, this value on iPhone 6Sis 6.66666667.

Define NORMAL as the normal force of touching the screen withoutproviding a force and it is a constant provided by the operation system.For instance, this value on iPhone 6S is 1.0.

Define NUM as the number of picker elements.

Calculate the minimum range value of the n-th picker element to beR1=NORMAL+(MAX−NORMAL)*(n−1)/NUM

Calculate the maximum range value of the n-th picker element to beR2=NORMAL+(MAX−NORMAL)*n/NUM

The range for the n-th picker element will therefore be any force valuegreater than or equal to R1 and less than R2.

For instance, in the iPhone 6S the maximum available force value is6.66666667. The normal touch force value is 1.0.

If the numbers picker needs to allow the user to pick an integer valuebetween 1 and 5 (i.e. 1, 2, 3, 4 or 5) then the force quantity valuescan be binned as binned force quantities. The force values ranges forthe binning will be as follows.

Value of 1 will match a force value greater than or equal to 1 and lessthan 2.133.

Value of 2 will match a force value greater than or equal to 2.133 andless than 3.266.

Value of 3 will match a force value greater than or equal to 3.266 andless than 4.4.

Value of 4 will match a force value greater than or equal to 4.4 andless than 5.533.

Value of 5 will match a force value greater than or equal to 5.533.

Although the change in force sensitivity defines the value picked by theforce picker, actually picking a value is a result of applying the samelevel of force for some period of time, i.e. a consistent hold.Therefore a time counter will begin counting the amount of time passedsince entering one of the ranges above. If the timer will hit apredefined period of time (e.g., 1 second) the currently matching valuewill be picked and the parent of the UI element will be notified aboutthe picked value (i.e., 1 to 5 in the example above). The UI element maybe modified based on the value. The predefined period of time can rangefrom any values, but will typically be between 0.5 and 2 seconds.

While this description mainly discusses the case of a number picker,this mechanism can also be applied to other types of pickers such assliders. A similar process may be used to control a webpage. The forcemeasurement may be applied to the user interface element to scroll thepage based on that force measurement. For example, a higher forcemeasurement may indicate to scroll further or faster.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the foregoing written description of the invention enables one ofordinary skill to make and use what is considered presently to be thebest mode thereof, those of ordinary skill will understand andappreciate the existence of alternatives, adaptations, variations,combinations, and equivalents of the specific embodiment, method, andexamples herein. Those skilled in the art will appreciate that thewithin disclosures are exemplary only and that various modifications maybe made within the scope of the present invention. In addition, while aparticular feature of the teachings may have been disclosed with respectto only one of several implementations, such feature may be combinedwith one or more other features of the other implementations as may bedesired and advantageous for any given or particular function.Furthermore, to the extent that the terms “including”, “includes”,“having”, “has”, “with”, or variants thereof are used in either thedetailed description and the claims, such terms are intended to beinclusive in a manner similar to the term “comprising.”

Other embodiments of the teachings will be apparent to those skilled inthe art from consideration of the specification and practice of theteachings disclosed herein. The invention should therefore not belimited by the described embodiment, method, and examples, but by allembodiments and methods within the scope and spirit of the invention.Accordingly, the present invention is not limited to the specificembodiments as illustrated herein, but is only limited by the followingclaims.

What is claimed is:
 1. An apparatus for interfacing with a computerdevice comprising: a display comprising a force sensor; a force sensorcontroller configured to determine a force input based on a touch inputto the display; and a processor coupled to the force sensor controllerand configured to: receive the force input, determine a user interfaceelement associated with the location of the force input, the userinterface element adapted to pick a value from among a plurality ofvalues, determine that a consistent level of force has been applied andhas remained within one of more than two ranges of force value for aperiod of time between 0.5 seconds and 2 seconds, and pick a value basedon the one range of force value within which the consistent level offorce has been applied and has remained for the period of time; whereineach value to be picked corresponds to one of the more than two rangesof force value, a minimum range value for each value to be picked isdetermined according to R1=NORMAL+(MAX−NORMAL)*(n−1)/NUM, and a maximumrange value for each value to be picked is determined according toR2=NORMAL+(MAX−NORMAL)*n/NUM, wherein R1 is the minimum range value, R2is the maximum range value, NORMAL is a normal force of a touch withoutproviding a force, MAX is a maximum possible force value of a touch, NUMis a number of the values to be picked, and n is an n-th value to bepicked.
 2. The apparatus of claim 1, wherein the processor is furtherconfigured to modify the user interface element based on the force inputby applying a numerical quantity to an input field of the user interfaceelement.
 3. The apparatus of claim 2, wherein the numerical quantitycomprises a binned quantity.
 4. The apparatus of claim 1, wherein theuser interface element comprises a picker.
 5. The apparatus of claim 1,wherein the user interface element comprises a scroll input.
 6. Theapparatus of claim 1, wherein determining a consistent level of forcehas been applied comprises converting the force input into a binnedforce quantity and determining that the binned force quantity has notchanged over a period of time.
 7. An electronic device comprising: adisplay; at least one sensor to detect a user interaction event with thedisplay; and a processing system to interpret data from the sensor andto display a user interface element on the display, wherein, the sensorreceives a touch input from a user and generates a digitized force valueand a display touch location, and wherein the processor receives thedigitized force value and display touch location from the sensor,determines if the user interface element has been activated based on thedisplay touch location, and if activated, determines that a consistentdigitized force value has been applied and has remained within one ofmore than two ranges of force value for a period of time between 0.5seconds and 2 seconds, and picks a value based on the consistentdigitized force value has been applied and has remained for the periodof time; wherein each value to be picked corresponds to one of the morethan two ranges of force value, a minimum range value for each value tobe picked is determined according to R1=NORMAL+(MAX−NORMAL)*(n−1)/NUM,and a maximum range value for each value to be picked is determinedaccording to R2=NORMAL+(MAX−NORMAL)*n/NUM, wherein R1 is the minimumrange value, R2 is the maximum range value, NORMAL is a normal force ofa touch without providing a force, MAX is a maximum possible force valueof a touch, NUM is a number of the values to be picked, and n is an n-thvalue to be picked.
 8. The electronic device of claim 7, wherein thedata input to the user interface element comprises a numerical quantity.9. The electronic device of claim 8, wherein the numerical quantitycomprises a binned quantity.
 10. The electronic device of claim 7,wherein the user interface element comprises a picker.
 11. Theelectronic device of claim 7, wherein the user interface elementcomprises a scroll input.
 12. The electronic device of claim 7 whereindetermining if the consistent digitized force value has occurredcomprises converting the digitized force value into a binned forcequantity and determining that the binned force quantity has not changedover a period of time.
 13. A method for modifying computer device userinterfaces comprising: receiving, from a display with an integratedforce sensor, a touch input; determining, by a controller operativelycoupled to the display, a display location and a force measurement basedon the touch input; mapping, by a processor coupled to the controller,the display location to a user interface element, the user interfaceelement adapted to pick a value from among a plurality of values,determining that a consistent level of force has been applied and hasremained within one of more than two ranges of force value for a periodof time between 0.5 seconds and 2 seconds, and picking a value based onthe consistent level of force has been applied and has remained for theperiod of time; wherein each value to be picked corresponds to one ofthe more than two ranges of force value, a minimum range value for eachvalue to be picked is determined according toR1=NORMAL+(MAX−NORMAL)*(n−1)/NUM, and a maximum range value for eachvalue to be picked is determined according toR2=NORMAL+(MAX−NORMAL)*n/NUM, wherein R1 is the minimum range value, R2is the maximum range value, NORMAL is a normal force of a touch withoutproviding a force, MAX is a maximum possible force value of a touch, NUMis a number of the values to be picked, and n is an n-th value to bepicked.
 14. The method of claim 13, wherein applying the forcemeasurement to the user interface element comprises inputting aquantitative value to an input field of the user interface element. 15.The method of claim 14, wherein the user interface element comprises apicker.
 16. The method of claim 13, wherein the user interface elementcomprises a scroll input.
 17. The method of claim 16, wherein applyingthe force measurement to the user interface element comprises scrollingthe page based on the force measurement.
 18. The method of claim 13,wherein determining a consistent level of force has been appliedcomprises converting the force measurement into a binned force quantityand determining that the binned force quantity has not changed over aperiod of time.